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Abstract: An Evaluation of the "Impact Assessment of the Training Programs" of a National 
Level Training Institution in India was conducted using the Kirkpatrick Method (KP Method). 
The studied Institution takes up research, provides training, offers consultancy and initiates 
action in the rural sector of India. The evaluation study used a detailed questionnaire for 
conducting a survey on the entire population of participants who attended the training 
programs in the selected study period. Personal interviews and workshops were also conducted 
with respondents to understand the behavioural changes and results seen in the work 
environment. The study brought out the need to understand the training evaluation as a 
continuous process, requiring periodic review and analysis of the needs of the various sectors 
of rural development. It reiterated the need to develop a systematic evaluation process within 
the institution. It also showed that the rural development professionals undergoing training 
themselves were keen to participate in the evaluation process so as to help in the process of 
self-learning and bring about sustainable changes. 
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1. Introduction 

There is much ado about training as an important component of the overall 
capacity building of any employee in an organization. Training is a planned process to 
acquire knowledge, skill, to modify attitude and/or behaviour. Training is done using 
different kinds of techniques, which include Lecture-cum-Discussion, Case Discussions, 
Group Discussions, and Exercises/Hands on Sessions, and Field visits. It is seen that sharing 
of learning experiences during training helps achieve a more effective performance in an 
activity or range of activities among the trainees. 

Efforts are made to evaluate training programs by both the training institution and 
the participating organizations, so that there can be an effective learning process. Where 
training is in-house the three most common reasons for evaluating the training program is; 
(1) to see how future programs can be improved; (2) to determine whether training should 
be continued and (3) to justify the existence of the training department itself. However most 
organizations continue to perceive training as a 'philanthropic' activity whose results is 
intangible in nature, and hence do not consider evaluation as a necessary activity. But over 
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the years, evaluation has become an important component as training requires enormous 
amount of investment in terms of human, financial and other resources. 

Donald Kirkpatrick in 1959 formulated the four Levels of Evaluation. The four levels 
represent a sequence of steps to evaluate training programs. Each level is important and has 
an impact on the next level. As one moves from one level to the next, the process becomes 
more difficult and time consuming, but it also provides more valuable information. 

Level 1 : Evaluation at this level measures how those who participate in the program 
react to it and is taken immediately after the completion of the training program. It can also 
be called a 'measure of customer satisfaction'. It is observed that a positive and favourable 
reaction from a few key persons in the group influences the future of a program. Infact a less 
than favourable reaction affects the motivation to learn among the participants. A negative 
reaction could greatly reduce chances of continuity for further programs. 

Level 2: A simple standardized paper and pencil test is administered (same test) 
before and after the programs as part of the evaluation process. This helps in understanding 
the extent to which participants change attitudes, improve knowledge and /or increase skill 
as a result of attending the program. 

Level 3: At this stage, it would be possible to assess the extent to which the 
behaviour is changed because of training and ideally assessed between six to nine months 
after the training is completed. 

Level 4: At this level the final impact results are taken into consideration. The final 
results can be in the form of increased production, improved quality, decreased costs, 
reduced frequency and/or severity of accidents, decreased costs, increased sales, reduced 
turnover and higher profits. It would help if the final objectives of the training program can 
be stated in these terms, for seeing improvement in the long-term. This assessment can be 
made between one to three years after completion of training, because otherwise there is a 
danger of 'lack of recall'. It is also that changes cannot always be singularly identified with 
the training received. 

The Kirkpatrick (KP) model has been used in different situations but predominantly 
in industrial settings, because, the final results are more quantifiable in nature both for the 
trainers and the trainees. This helps the authorities to take decisions about the continuation 
of training in a very proactive manner. Some of the companies which have used this 
evaluation method very rigorously include Motorola Corporation, University of Wisconsin, 
USA which evaluated the training for developing supervisory skills in the staff, and Intel 
Corporation which used the method to evaluate a corporation wide performance 
improvement system (Kirkpatrick 1998). 

Increasingly, the KP model is being used by training organizations to understand 
the impact of the training programs even where the results are not very tangible in nature 
(Marcotte, Bakker-Dhaliwal and Bell, 2002). 

The present paper is based on an evaluation study conducted for a training 
institution in India. It brings forth the method in which the training programs were evaluated 
following the four levels of the KP model. The studied institution over the years has emerged 
as a premier training and research institute in the field of rural development in India. 
Training rural development professionals is complex as it encompasses many areas of 
knowledge and myriad skills which are sought to be transferred through various methods. 
The composition of the participants is heterogeneous in nature. The studied Institution 
imparts training mainly to lower and middle level government officials working in the 
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Departments relating to rural development and related sectors, which include Water Supply 
and Sanitation, Health, Local Self Governments, Public Works Departments, Roads and 
Buildings, among others.. The personnel are nominated by the respective departments for 
training every year based on the proposed internal requirements. 

The training institution went in for the evaluation study to assess the impact of the 
training programs and get a macro picture across all the training programs and all 
participants across the country. It covered a three year period for the survey. The main 
objectives of the study were: 

(i) to understand the impact of training on the participant's knowledge, skills and attitudes; 

(ii) to understand whether the training programs had affected participant's performance in 
the work environment back home and (iii) to understand the efficacy of institutionalizing an 
evaluation process using the KP model. 

The paper is divided into three sections. The second section gives the methodology 
of the study based on the KP model, the third section discusses the survey results with respect 
to the four levels of the KP model and the last section gives the conclusions drawn from the 
study. 

2. Methodology 

A primary survey was done using a detailed questionnaire as a tool. The survey 
helped in establishing an understanding of all the four levels of evaluation - reaction, 
learning, changes and results. The survey used the entire population of participants who 
attended the training programs of the Institution over the selected three years. The 
institution on an average trained 3000 participants every year from across the country in its 
TOO training programs per year. 

The questionnaire had three main parts - 

I. Personal details - to build the profile of the participants; 

II. 'Effectiveness of Program' was studied with key questions on whether the objectives 
of rural development were met within the program. The participants were asked to 
rate the program content and design on the basic inputs of knowledge, skills and 
attitudes. 

III. 'Professional relevance of training' was evaluated with key questions asking how 
relevant the program content was for meeting the local needs and whether there 
was enough practical application which could be used for working or transferring the 
knowledge to functionaries further down the line. It also probed whether the 
learning could be shared with other colleagues in the organization and lastly 
whether the course had helped in the organizational performance. 

Database: 

In the first instance, the database of 9000 participants was cleaned for missing 
names and incomplete addresses. The questionnaire was then posted to all the participants 
together with a stamped self-addressed envelope. Three reminders were also posted over a 
period of three months to the trainees who had not replied. Questionnaires were also posted 
to e-mail ids wherever available. The replies received were tabulated in the SPSS format 
and analysed. 
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The exercise with the database highlighted the weakness in the system where the 
names of all the participants of the programs, names of the resource faculty and feedback 
received were not maintained properly. This seemed to have happened because of a lack of 
a central system to maintain records of the various programs. The institution, recognizing 
this lacuna, had recently developed a new digital format to capture the information 
correctly. With this, a good electronic database would be in place which while giving a clear 
picture of all training programs, could be used as a ready input for any future evaluation. 

The survey received a 1 6% response rate from a total population of 9000 
participants. In the second phase of the study, personal interviews were also conducted to 
understand the perceptible change in knowledge, skills and attitudes after the training, for 
Levels 3 and 4. 

Further the survey findings were shared in four regional workshops held in the 
north, east, west and southern parts of India respectively. A background paper was prepared 
with a regional focus, and circulated among all participants of the workshop. The workshops 
helped the study team to understand the impact of training programs and the future 
changes and interventions that could potentially improve the reach and relevance of training 
programs for rural development. The workshops also helped validate the findings of the 
survey and obtain suggestions for improving the quality of the training programs. 
Suggestions were also received about new ways for evaluating training programs in future. 

3. Survey Results 

Profile of the participants: 

The survey had a response of 87% from men when compared to 13% from women. 
This result was clearly in conformity to the pattern of the database. Fewer women were 
trained as compared to men. The respondents were mostly in the age group of 41-50 years. 
The educational profile however showed a very interesting insight across the country, that 
the majority of the trainees were post-graduates. 

Majority of the responses came from those participants who were trained in the 
most recent year of the study period. This confirmed the Kirkpatrick model, which states that 
the evaluations conducted within a shorter span of time after the actual training has been 
given, is more relevant. The rate of recall reduced as time passed by. 

The survey showed that the majority of the participants trained were those having 
experience for about 10 years or more in the rural sector. Another very interesting fact 
emerged from the study that nearly 70% of those trained remained in the same position 
after receiving the training. 

Level 1 

In the studied institution, feedback was taken at the end of every program on important 
heads such as realization of program objectives, program content, and training methods on 
a 5-point scale. This is the Level 1 process of the KP model. The institution over the years 
had continually received a rating of over 84% at this level, which reflects the quality of 
performance of training activities carried out by the Institute. 

The survey also sought to understand the rating of the programs. The participants 
were asked to rate the program content and design, teaching methodology and style, 
material used and distributed on a 3-point scale - (a) innovative and effective, (b) repetitive 
but effective, and (c) ineffective. 
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Figure 1. Rating of Programs - Innovative and Effective 

The survey showed that nearly 55% of the respondents observed that overall the 
training components were innovative and effective. But when seen closely, program design 
stood first followed by program content, as shown in Figure 1 . All the other three attributes, 
namely, teaching style and delivery, teaching methodology and the material used and 
distributed were ranked lower showing that there was much scope for improvement in these 
areas. 37% of the total respondents said that the programs were repetitive but effective and 
the rest said that they were ineffective. 

The survey reaffirmed the initial reaction of the participants even after a time gap 
of three years. However the respondents mentioned that more could be learnt through 
interactions with the trainers. Clearly there was a need for "hands-on" knowledge of 
resolving issues on ground through local case studies. It was also important for the 
respondents that the lead coordinators or trainers were available throughout the period of 
the program. 

Another important question under the "effectiveness of the program", was the 
"extent to which objectives of rural development" were achieved in the various training 
programs. The respondents on a scale of 1 to 5 measured the objectives given below, with 5 
standing for 'completely' to 1 standing for 'minimally'. 

^ Understanding of concepts, definition, planning, and implementation of rural 
development programs; 

^ Awareness of personal and institutional role in Rural Development; 

^ Bringing about attitudinal change towards Rural Development; 

^ Develop new skills for a sustainable approach to Rural Development; 

^ Appraisal of rural development projects; 

^ Understanding institutional mechanisms; 

^ Analysis of strengths/ weaknesses of various anti poverty programmes; 

^ Role of Information Technology in Rural Development. 
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Figure 2. Objectives of Rural Development Achieved 

For simpler analysis, the scales of 4 and 5 marked by the respondents against each 
of the objectives were taken together, and similarly 1 and 2 were clubbed to understand 
minimal achievement. The scores in Figure 2 shows the percentage of respondents who felt 
the respective objective was achieved on the five point scale. 

The survey showed that the training programs were able to achieve a near sense of 
completeness of objectives (Figure 2) in terms of 'understanding of concepts'- as shared by 
74% of the respondents across the various categories of trainees, followed by 'awareness of 
role in Rural Development' and 'attitudinal change towards Rural Development', clearly 
shown by the percentage scores. 'Development of new skills' clearly received a lower 
ranking. 


Level 2: 

Measurement of 'learning' at Level 2 requires administration of tests at the 
beginning and the end of training programs. This was not taken up in the study because: (1) 
the institution had not conducted such tests for the training programs being evaluated and 
(2) the tests could not be conducted as part of the survey as the training programs being 
evaluated were spread over a period of three years in the past. Level 2 measurement is 
useful where the training programs predominantly target conceptual understanding or 
transfer of skills. As one of the objectives of the study was to look at the need for a regular 
evaluative process, the need for a Level 2 assessment was discussed with the participants in 
the workshops. 
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It came out clearly through the interviews that the respondents were open to having a test 
on concepts both before and after the training program. The respondents even suggested 
that the participant's performance should be assessed during the course of the program for 
both knowledge and skill inputs using a simple questionnaire. It was strongly recommended 
that the results of the test should be communicated directly to the head of the department 
where the participant worked. In cases where the results could not be shared on a one to 
one basis with the participants, it was suggested that the institution could think of having a 
collective group grading system both pre and post training. 


Level 3 

The questionnaire had a separate part on "Professional relevance of training" 
which tried to understand the Level 3 and Level 4 attributes of change and results of 
training. The components included the relevance of Program content, extent of practical 
application of the program, benefits of training, ability to share the information received and 
change in organisational performance. 

The survey results were rechecked through personal interviews conducted with 16% 
of the total respondents spread across the four regions of the country. The focused interviews 
helped in understanding whether the participant observed any perceptible change in his/her 
knowledge, skill, and attitudes after the training program. The trainee was also asked to 
identify how the learning could be utilized in the work situation. 

The survey clearly showed that the program content had high relevance. But in 
comparison to the extent of practical application of the program, content showed medium 
relevance. As the participants were heterogenous in nature, hailing from different parts of 
the country and sometimes working in different departments, they could not implement what 
they had been trained in. The most common reasons for this shortcoming was that; (i) the 
training was not always relevant to the specific areas of work, (ii) tools and techniques 
(information technology and GIS) shown during the training did not exist in the work 
environment, and /or (iii) the training program did not cover issues of 'how to resolve the 
problem' which arose in the field and (iv) after training, the trainee was shifted to another 
area of work, even if working in the same department. 

With regards to the benefit of training, it is interesting to note that the respondents 
had an increased awareness through the program. The respondents were asked to rank the 
benefits of training, on a scale of 1 to 5, with 5 standing for 'extremely useful' to 1 for 'not at 
all useful'. The benefits being: 

1 . Developing concern for rural development 

2. Improvement in inter personal relations 

3. New concepts / ideas introduced 

4. Networking 

5. Increased efficiency in delivery of service 

6. Improved service levels and 

7. Efficient utilization of financial resources 
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Figure 3. Benefits of Training 

Here again the scales of 4 and 5 and scales of 1 and 2 were put together for 
simpler understanding. The Figure 3 shows that 74% of the respondents said that the 
greatest benefit that they had derived from the training program was in 'developing concern' 
for rural development. The other two main benefits were seen in improvement of inter- 
personal relations and learning of new concepts/ ideas introduced in the field of rural 
development. 

It was observed that the changes and the ensuing results in the organizational 
context were the more difficult steps of evaluation. No final results could be expected unless 
a positive change in behaviour occurred. It was also seen that at times, the change was not 
fully under the control of the participant. Participants could not change their behaviour 
unless they had a chance to do so. Also, it was impossible to predict when change occurred, 
and sometimes, the participant may have decided against the change after contemplation 
because (s)he felt the earlier style was better or that the environment was not conducive for 
change. Infact it was seen that in many cases, changes could not be singly attributed to the 
training received. 

At such times it became imperative to see how the participants came to be 
nominated for the training program. From the survey it came out that the most common 
reasons were: (i) the participants considered themselves to be the most 'Suitable person' for 
training. But most could not elaborate further on this. Under the circumstances, it could not 
be inferred further whether any specific mode of selection was identified or was in place 
within the departments when identifying the suitable person; (ii) Nominated - This in most 
cases meant that the participant had not really thought about his/her nomination. The 
trainees had attended because they had been sent for training. From the trainer's point of 
view, it seemed that many times the most 'dispensable' person was sent for training. This 
was true in cases where the nominations were received in the last minute; (iii) As per 
procedure - this was also a reason similar to that of the suitable person, where the trainees 
could not clearly define what the procedure was for being selected to the training program; 
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and (iv) Don't know - the trainees did not simply know why they had been sent for training, 
even after a lapse of nearly 1 to 3 years. 

This seemed to give a clear picture of how training was not considered serious 
business by the sponsoring authorities. The point was reiterated during interviews with the 
Heads of the Departments. The Departmental heads many times were trying to use the 
budget allotted under capacity building. No clear guideline had been followed before the 
nominations were made for the training program. 

It was seen that the impact of the training was best seen where the needs of the 
training was clear right from the beginning. But under the circumstances, one could venture 
to say that the impact of the training was limited by the participant who was nominated. 
Because of this the studied institution was compelled to design a more common, base level 
program catering to a heterogenous group rather than a specific issue based training 
program. 

All the same, once selected for training, the respondents (77%) came with some 
expectations before the training, though they were not able to clearly express what their 
expectations had been. The respondents (75%) also went on to say that their expectations 
had been met. 23% of the respondents however said that the question of expectations being 
met was not applicable to them, as they had not really understood what they wanted initially 
from the program nor realized the objectives of the program even by the end of the training. 

59% of the participants had become aware of new needs and the personal 
handicaps in terms of skills required for meeting the needs after attending the training. Such 
participants wanted to attend repeat programs at an advanced level. In some cases they 
even wanted to attend a few basic courses for better understanding of the concepts. 
Research Methodology and Communication Skills were two such popular programs. 

When asked about the 'perceptible changes' in knowledge, skills and attitudes after 
the training, 99.5% said they had observed that there was some learning, though unable to 
pinpointedly mark it in their work. 

Overall it came out that the training programs had created awareness, and helped 
them in 'being more concerned' about rural development. It was found that the middle level 
functionaries working in the field, who were working very grudgingly earlier, realized that 
they could contribute positively in the work environment. Training had helped them become 
aware of the problems and needs of rural development. And the already 'aware' participants 
became more articulate in their expression. The training helped them in thinking in a more 
sequential manner to help find solutions locally. 

These findings directly related to the training programs which had good program 
content, and design, but was not sufficiently performance oriented in terms of practical 
application. This perhaps could be one of the main reasons for the participants inability in 
realizing a perceptible change, or in they having a continued niggling feeling that something 
was missing, which restrained them from completing the 'last mile' successfully. 

This only confirmed that in the training institution, the trainer would have to take 
note of the work situation that the participant returned to while designing the course. And 
for a more effective result, an enabling atmosphere would need to be created in the 
Departments where change was expected. 
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Level 4 

For Level 4, the study tried to get the respondent to quantify through figures, the 
sharing of learning in terms of meetings held, number of beneficiaries influenced and such 
others. The Heads of Departments were contacted to get their views on what was needed for 
rural development. 

When asked about how the knowledge had benefited the organization, 
respondents found it difficult to quantify the benefits derived. 96% of the respondents were 
emphatic that they had utilized the learning from the training in their work situation. But a 
large number, were unable to explain how it had been translated into work. It was seen that 
only 68% said that they had actually attempted knowledge sharing while the rest had made 
no attempts in this direction. 

The common ways in which the participants utilized the learning were (i) to 
implement day-to-day work and prepare reports, planning and evaluation, (ii) design and 
implement new schemes effectively and (iii) Learned skills to train the field staff. 

One of the main reasons why no quantification could be made, because there were 
no attempts prior to the training or later to document the work done to help fill in the gaps 
where needed. Hence no comparative analysis could be done. 

However the survey results under the section 'Professional relevance' of the 
program helped gain insight into this aspect. The question was administered on a 5-point 
likert scale. The results showed a definite change among the trainees in (a) new concepts / 
ideas were introduced, (b) increased productivity, (c) networking with different people, (d) 
increased efficiency in delivery of services, (e) improvement in interpersonal relations, (f) 
improved service levels and (g) career advancement. However reduction of execution time 
of projects and improvement in revenue/project viability got relatively low scores, meaning 
perceptible changes could not be seen in these two parameters. It can be seen that these 
heads are such which need a systemic organizational change with more awareness among 
all concerned. Infact it was interesting to note that many respondents shared the view that 
their 'bosses' and Heads of Departments needed to be trained for a more systemic change 
and overall better performance in the organization. 

In the current scenario, the departments in Government had no regular process 
whereby the learning could be transferred to the work environment. Most times the back 
home work environment remained neutral with no expectations from the Departments. 

The respondents also suggested ways to improve the training programs so that 
more changes could be seen at Level 3 and Level 4 in the long run, which were as follows - 

1) Most participants were unaware of the gamut of training programs offered by the 
Institution. It was suggested that if the calendar of the training programs were 
communicated to the concerned Department a year in advance then the participants 
could choose those training programs which would be most relevant to them. 

2) Selection of participants had to be done in an objective manner by the sponsoring 
department, which was sending the participants; based on the relevance of the 
training program. Younger personnel with more years in service would have to be 
sent for training. 

3) The training Institution would have to scrutinize the nominations sent, through its 
own rigorous protocol for selection of the participants. This way the right participant 
who was well experienced in a relevant field would get enough inputs for 
betterment. 
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4) Every group of participants needed a program guide from the parent Department to 
help them through the training program. 

5) Impact of training programs to be evaluated within 3-6 months after training 
program to help reiterate the learning. 

4. Conclusions 

Overall, the studied institution had delivered through its numerous training 
programs. The participants were happy about the training they had received, and looked 
forward to learning more from the Institute. There were however shortcomings in trainings 
in terms of the nominations of the participants, and the problems faced by the participants 
in utilizing the learning in their home environment. 

It was interesting to note that through the study, the respondents came up with the 
same kind of suggestions for evaluation as suggested under the KP model, even when the 
study team had never mentioned that the study was using such a model for analysis. 

From the study it came out quite clearly that: 

a) Level 1 is the best evaluation method of the four levels with respect to cost and efficiency. 

Level 1 evaluation was being done at the institution and needed to be continued. 

b) Level 2 was presently not done at the institution. The various respondents were keen on 

being evaluated during training as also at the start and the end. 

c) The Level 3 evaluation clearly brought out that there were perceptible behavioural 

changes in the respondents after training, though not measurable in all cases. The study 
clearly showed that nearly 70% of those trained remained in the same position after 
training, which offered great hope for active policy interventions and changes that could 
be brought through the training programs. 

d) For Level 4, sponsoring departments had to clearly put down the expected results. The 

study showed that there was need for cooperation between sponsoring authorities and 
the trainers to design programs which could become more functional and useful for 
changes to happen at the ground level. The respondents were clearly keen to cooperate 
and take the efforts forward. 
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